An Investigation of Grid Performance Predictions Through Statistical Learning

Scheduling in large scale distributed systems like Grids [13] is challenging because of their dynamics, heterogeneity, and lack of central control. In such environments performance predictions play an important role on providing input information required by a Grid scheduler. In this paper we investigate a set of statistical learning techniques in predicting two performance metrics in the Grid, namely, application run time and queue wait time on space-shared computing resources. We propose Instance Based Learning (IBL) as a common framework to predict both metrics, with the IBL parameters optimized by genetic search. Bias-variance analysis of error is conducted on tuning parameters on subsets of training data (local tuning) as well as on the whole set (global tuning), based on which a method is developed to select the tuning type adaptively. An efficient search tree structure called “M-Tree” is also integrated into our algorithm to speed up k-nearest neighbor search. Experimental studies are conducted to evaluate our methods using real world workload traces, which are collected from the NIKHEF cluster on the LHC Computing Grid and Blue Horizon in the San Diego Supercomputer Center. The results show that acceptable prediction accuracy is achieved using the proposed techniques, and the performance is able to meet the real time requirements in the Grid scheduling process.